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culturally appropriate prevention materials; 2) reduce ATOD use and 
violence-related behaviors among Native Hawaiian youth by providing direct, 
culturally appropriate educational services at high-need schools and 
communities; and 3) build the capacity of schools and communities to provide 
safe and drug- free activities for Native Hawaiians. These goals were refined 
into 10 measurable objectives. Results show that the NHSDFSCP met three of 
its eight objectives for which data were available. The objectives that were 
not fully met included increased knowledge of the dangers of ATOD use, 
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1999-2000 Evaluation of the 
Native Hawaiian Safe and Drug-Free 
Schools and Communities Program: 
End of Year Report 



Executive Summary 



This report provides an external evaluation of the Native Hawaiian Safe and Drug-Free Schools 
and Communities Program (NHSDFSCP). The NHSDFSCP is currently in the fourth year of a 
funding cycle that began in 1996. This report is based upon data from the entire program period 
March 1, 1999 - February 28, 2000, and is meant to serve as a comprehensive report to augment 
the three-quarter report completed by Pacific Resources for Education and Learning (PREL) in 
January 2000. 

The purpose of this report is to answer a fundamental question: 

Did the program achieve its stated measurable goals and objectives? 

Further, the report will present evidence on the effectiveness of each of the major program 
components and make commendations and recommendations for future program development. 

In summary, data were available on eight of ten program objectives as of the end of the 1999 - 
2000 program year. The results show that the NHSDFSCP met or partially met three of its eight 
stated objectives for which data were available. The objectives that were not met included 
increased knowledge of the dangers of Alcohol, Tobacco, and other drugs (ATOD) use, by those 
receiving materials only and those who participated in educational activities; intended 
involvement in violent behavior; increased cultural awareness; and increased knowledge of the 
dangers of violence-related behaviors. However, the findings for these objectives fell between 
2% and 8% of the 80% targets. It may be concluded that the program achieved its desired 
outcomes and met its three overall goals. 

The goals and related outcomes were further specified in the component evaluation sections. The 
goals and activities of the NHSDFSCP reflect its alignment with the strategies promulgated by 
the Center for Substance Abuse Prevention (CSAP). Although the program carries out some 
activities under all six of the CSAP strategies, previous evaluations recommended that the 
program emphasize three components. These three strategies effectively define the essence of 
the NHSDFSCP. 



• Information Development/Dissemination 
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• Education 

• School/Community-Based Processes 

A review of program activity form (PAF) data gives an overview of program-related activities in 
terms of the six Center for Substance Abuse Prevention (CSAP) strategies, as well as in terms of 
other relevant variables. The relative proportion of persons served related to each of the six 
components is illustrated. They provide support for the program emphasis on Information 
Development/Dissemination, Education, and School/Community-Based Processes and 
Approaches. In total, these three components represent services provided to 99 percent of all 
involved with the program. 

In addition to the PAF data, a “Milestone Achievement Analysis” was conducted to compare 
planned program activities with their actual implementation. By the end of the current funding 
cycle the program anticipates the completion of 27 of 35 planned activities (77%). Another five 
activities (14%) are either ongoing or partially completed, two (6%) will be carried over, and one 
(3%) has been cancelled. 

The results for each of the three major components are presented and are summarized below: 
Conclusions: Information Develovment/Dissemination 

As of the first three quarters of the program year, NHSDFSCP has met its desired outcomes for 
this component. This is one of the three components receiving the most program emphasis. 
Through this component, the program is responsive to a frequently expressed need in the schools 
and communities served for prevention materials that are culturally appropriate for the youth of 
Hawai‘i. The NHSDFSCP researches, produces, and makes widely available materials that are 
particularly appealing to youth of Native Hawaiian ancestry. Other materials are also made 
available. The increasing numbers of such materials in circulation and the increasing use of the 
resource libraries are a testament to the effectiveness of the program in the area of information 
development and dissemination. 

Conclusions: Education 



Education remains the cornerstone of the NHSDFSCP. The workshop evaluation forms provide 
evidence that the participants feel that the experience had an impact on them, on their knowledge 
and beliefs, and on the decreased likelihood of their being involved in both alcohol and other 
drugs and in violent behaviors. The E Ola Potto evaluation shows preliminary evidence of 
impact on students’ knowledge of Hawaiian concepts and values as well as reduced involvement 
in violent behavior for elementary school students. 
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Conclusions: School/Communitv-Based Processes and Approaches 



This component represents a major strategic thrust of the NHSDFSCP. The program is very 
effective at collaboration with similar agencies and within specific communities. This affiliative 
approach is both cost efficient and highly valued within the Hawaiian social system. This 
component has also stimulated the funding of research upon which to rationally develop and 
apply program plans. Finally, in terms of increasing the knowledge and skills of teachers, those 
who have been the direct recipients of curriculum training are overwhelmingly favorable about 
the training and its potential for having a positive impact on students. 

Commendations and Recommendations 



Commendations 



1 . The NHSDFSCP Director is commended for her firm commitment to sound program 
planning based on outcome data and needs assessment results. She has maintained the use of 
the Logic Model approach to program and evaluation planning. Each year the program has 
been able to set clearer outcome objectives, and this year they were able to meet three out of 
eight of those for which data were available at the end of the program period. The ones not 
met were within 8 percentage points of criterion. This is impressive in that baseline rates for 
these particular outcome targets had not been previously established. 

2. The program is commended for its application of needs assessment information. From the 
beginning the NHSDFSCP has been guided by continuous needs assessment efforts. These 
have included monitoring Native Hawaiian student and adult involvement in alcohol, 
tobacco, and other drug (ATOD) use and violence as well as community-felt needs. Needs 
assessment provided the information that allowed the program to focus on those schools and 
communities with a high number and proportion of Native Hawaiians. 

3. One of the first clearly defined needs was for information on ATOD use and violence to be 
presented in forms that were culturally sensitive and locally relevant. Accordingly, the 
program spent much time in developing new materials based upon Hawaiian or local cultural 
themes. The program is commended for the cultural relevance of these materials. 

4. The Alcohol and Drug Abuse Division notes that “improvements in alcohol and drug 
education must occur at both the school and family level” (Klingle & Miller, 1999, p. 18). 
Consistent with this recommendation, the NHSDFSC program is commended for its 
development of a school-based version of their E Ola Pono program that includes parents. 

5. A related recommendation by the Alcohol and Drug Abuse Division is that “comprehensive 
school-based substance abuse prevention programs must begin no later than the fourth grade” 
(Klingle & Miller, 1999, p. 27). The NHSDFSC program is emphasizing development of its 
program for fourth graders and is developing “boosters” against substance use to occur 
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thereafter at strategic points through middle and high school and is commended for this 
initiative. 



Recommendations 



1. It is recommended that the program carry out a more intensive, observational assessment of 
the E Ola Potto curriculum. This should be done using program developers as teachers or in 
direct support of teachers so that they have a better on-site sense of the implementation of the 
curriculum. 

2. Now that the E Ola Potto curriculum has been pilot-tested and refined, it is recommended 
that the program initiate a longitudinal study of its long-term effects on students. The 
limitations of a one-semester intervention and pre and post measurements are obvious. 
Whether the program provides long-term protective or preventive benefits can only be seen 
by following carefully constructed samples of students through their high school years - 
possibly even beyond. Long- term outcomes to be followed might include school attendance, 
grades and achievement, and participation in extra-curricular activities. 

3. The effect of the E Ola Potto curriculum should be bolstered by a greater repetition of its 
component lessons. This repetition might take the form of a longer intervention or periodic 
booster sessions to critical program features following initial exposure to the curriculum. 

4. A need for instrumentation revision was evident after the pre-test data collection period. 
Teacher interviews revealed unanticipated problems with the instrument including its length, 
associated student fatigue and apathy, and question ambiguity. A revised instrument was 
pilot-tested in the summer during a summer school application of the E Ola Potto 
curriculum. This instrument, complete with new and revised scales, is an improvement over 
the previous one, but is not yet finalized. Future instrumentation improvements should 
attempt to ensure that assessments are culturally appropriate and understandable, given 
existing literacy limitations of students. 

5. A further recommendation for the E Ola Potto evaluation is the use of qualitative research of 
the curriculum. For example, there should be the inclusion within the overall research design 
of more exploratory and qualitative data, such as anecdotal reports, teacher checklists, and 
non-participant observations. 

6. The trend data provided by this study provide information regarding the effects of maturation 
on the incidence of gateway drug experiences, and normative perceptions of gateway drug 
use. This type of information should be applied toward program needs assessment and for its 
application toward targeting intervention populations. 

7. A recommendation made by the Alcohol and Drug Abuse Division is to widely distribute 
information about the availability of treatment and counseling services. Through its 
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Information Development and Dissemination component, the NHSDFSC program should 
continue to inform students about where and how they can seek help for substance abuse. 
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1999-2000 Evaluation of the 
Native Hawaiian Safe and Drug-Free 
Schools and Communities Program: 
End Of Year Report 

Introduction 



This report provides an external evaluation of the Native Hawaiian Safe and Drug-Free Schools 
and Communities Program (NHSDFSCP). The NHSDFSCP is currently in the fourth year of a 
funding cycle that began in 1996. This report is based upon data from the entire program period 
March 1, 1999 - February 28, 2000, and is meant to serve as a comprehensive report to augment 
the three-quarter report completed by Pacific Resources for Education and Learning (PREL) in 
January 2000. 

The purpose of the evaluation of NHSDFSCP is to answer a fundamental question; 

Did the program achieve its stated measurable goals and objectives? 

This report will present evidence on the effectiveness of each of the major program components 
and make commendations and recommendations for future program development. 

The program identified a set of overall measurable objectives based upon its goals emphasizing 
student and participant outcomes. This evaluation focuses upon these outcomes. 



Did the Program Achieve its Goals and Objectives? 



The goals of the NHSDFSCP are as follows: 

I. Increase knowledge of the dangers of Alcohol, Tobacco, and other Drugs (ATOD) use, 
and of violence, and increase cultural awareness by increasing the availability and use of 
culturally appropriate prevention materials. 

II. Reduce ATOD use and violence-related behaviors among participating Native Hawaiian 
youth by providing direct, culturally appropriate educational services at high-need 
schools and communities. 

III. Build the capacity of schools and communities to provide safe and drug-free activities for 
Native Hawaiian youth and adults. 
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These goals align with the three major program components: Information 
Development/Dissemination, Education, and School/Community-Based Processes. 

The program’s desired outcomes are as follows: 

1 . Increase knowledge of the harmful effects of ATODs. 

2. Increase knowledge of the harmful effects of violence. 

3. Maintain or decrease use of ATODs. 

4. Maintain or decrease student involvement with violence. 

5. Increase protective factors as indicated by student participation in school. 

6. Increase Hawaiian cultural awareness. 

7. Increase skills of those working in the prevention field. 

These desired outcomes were further refined into a set of measurable objectives. Table 1 
provides the objectives along with findings using data available at the completion of the program 
year. 



Table 1. Program Objectives and Findings 



Objectives 


Findings 


Objective 

Achieved? 


1 . At least 80% of those receiving 
materials will report an increased 
knowledge of the dangers of 
ATOD use. 


Of 42 persons surveyed, 30 (72%) 
gave ratings of “Agree” or “Strongly 
Agree.” 


NO 


2. At least 80% of participants in 
school/community-based 
educational activities will report 
an increased knowledge of the 
dangers of ATOD. 


Of 239 respondents, 184 (77%) gave 
ratings of “Agree” or “Strongly 
Agree.” 


NO 


3. At least 80% of those receiving 
materials will report an increased 
knowledge of the dangers of 
violence-related behaviors. 


Materials investigated by Tracer 
Study did not address violence issues 


N/A 


4. At least 80% of participants in 
school/community-based 
educational activities will report 
an increased knowledge of the 
dangers of violence-related 
behaviors. 


Of 249 respondents, 183 (73%), gave 
ratings of “Agree” or “Strongly 
Agree.” 


NO 



BEST COPY AVAILABLE 
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Table 1. Program Objectives and Findings (cant’d) 



Objectives 


Findings 


Objective 

Achieved? 


5. At least 80% of participants in 
educational activities (E Ola 
Pono; workshops) will show a 
level or reduced ATOD use or 
intended use. 


Of 68 E Ola Pono participants, 90%, 
93%, and 99% reported level or 
reduced use of alcohol, cigarettes, and 
marijuana, respectively. 

Of 762 respondents, 571 (75%) gave 
ratings of “Agree” or “Strongly 
Agree.” 


Ygg 

NO 


6. At least 80% of participants in 
educational activities (E Ola 
Pono-, workshops) will show a 
level or reduced involvement in 
violence-related behaviors. 


Of 68 E Ola Pono participants, 77%, 
76%, and 75% reported level or 
reduced engagement in, observations 
of, and fears of violence. 

Of 202 respondents, 157 (78%) gave 
ratings of “Agree” or “Strongly 
Agree.” 


NO 


7. At least 80% of participants in 
educational activities will show 
an increase in resiliency as 
indicated by participation. 


Relevant attendance data will be 
available after the end of school year 
1999-2000. 


N/A 


8. At least 60% of those receiving 
materials will report increased 
cultural awareness. 


Of 42 respondents, 3 1 (74%) gave 
ratings of “Agree” or “Strongly 
Agree.” 


YES 


9. At least 80% of participants in 
educational activities will show 
increased cultural awareness. 


Of 786 respondents, 582 (74%) gave 
ratings of “Agree” or “Strongly 
Agree.” 


NO 


10. At least 80% of participants in 
school/community-based 
educational activities will report 
increased skills in prevention. 


Of 162 respondents, 125 (77%) 
reported an increased ability to 
prevent substance abuse. Of 23 1 
respondents, 165, or 71% reported an 
increased ability to prevent violence. 


YES 



In summary, data were available on all but two of the program’s stated objectives as of the end of 
the program year. The results show that the NHSDFSCP met or partially met three of its eight 
stated objectives for which data were available. The objectives which were not fully met 
included increased knowledge of the dangers of ATOD use, intended involvement in violent 
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behavior, increased cultural awareness, and increased knowledge of the dangers of violence- 
related behaviors. However, the findings for these latter objectives fell between 2% and 8% of 
the 80% targets. 

It may be concluded that, overall, the program achieved its desired outcomes and met its three 
overall goals. The goals and related outcomes will be further specified in the component 
evaluation sections that follow. 
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Component Evalnation 



Overview 



The goals and activities of the NHSDFSCP reflect its alignment with the strategies promulgated 
by the Center for Substance Abuse Prevention (CSAP). Although the program carries out some 
activities under all six of the CSAP strategies, previous evaluations recommended that the 
program emphasize three components. These three strategies effectively define the essence of 
the NHSDFSCP. 

• Information Development/Dissemination 

• Education 

• School/Community-Based Processes 

A review of program activity form (PAF) data gives an overview of program-related activities in 
terms of the six components as well as in terms of other relevant variables. The relative 
proportion of persons served related to each of the six components is illustrated in Table 2 and 
Figure 1 . They provide support for the program emphasis on Information 
Development/Dissemination, Education, and School/Community-Based Processes and 
Approaches. In total, these three components represent services provided to 99 percent of all 
involved with the program. 



Table 2. Number of Persons Served by Program Component 



Program Component 


Persons Served 


Percentage 


Information Development 


2,278 


17% 


Education 


6,369 


47% 


Alternative 


15 


0% 


Problem Identification 


15 


0% 


School/Community-Based 


4,703 


35% 


Environmental 


59 


1% 


Total 


13,439 


100% 



BEST COPY AVAILABLE 
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School/Community 
Based 
35% 



Environmental 

1 % 



Information 

Development 

17 % 




Problem 
Identification 

0% Alternative 

0 % 



Education 

47% 



Figure 1. Percentage of Persons Served by Major Program Component 



Whom does the program serve? Most activities are directed towards students (see Table 3 and 
Figure 2). In addition, the activities serve an audience that is primarily Native Hawaiian or part- 
Hawaiian (see Table 4 and Figure 3). 



Table 3. Number of Persons Served by Role 



Role 


Service Hours 


Percentage 


Student 


9.653 


72% 


Teacher 


848 


6% 


Parent 


561 


4% 


Service provider 


1.682 


13% 


Other 


655 


5% 


Total 


13.399 


100% 



Ierjc 
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Table 4. Number of Persons Served by Ethnicity 



lEthnicitv 


Particioants 


Percentage 


Hawaiian 


9.429 


70% 


INon-Hawaiian 


3.970 


30% 


llotal 


13.399 


100% 
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The next Figure (4) and Table (5) present the statewide distribution of activities for Native 
Hawaiians. The Kamehameha Schools (KS) programs, which serve KS and non-KS students 
statewide, Hawai'i, and Leeward districts received the most service hours because the program 
targets areas of high Native Hawaiian concentration and these areas serve the highest percentages 
of Native Hawaiian children and adults. 



Table 5. Number of Service Hours by School District 



School District 


Service Hours 


■ Percentage 


Hawaii 


5,180 


13% 


Honolulu 


2.096 


5% 


Kauai 


2,473 


6% 


Leeward 


3.230 


8% 


Maui 


1.508 


4% 


Windward 


5.881 


16% 


Statewide 


3.508 


9% 


KS 


14.084 


37% 


Other 


876 


2% 


Total 


38.836 


100% 
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Figure 4. Proportion of Service Hours by School District 



In addition to the PAF data, a “Milestone Achievement Analysis” was conducted to compare 
planned program activities with their actual implementation. Full results are contained in 
Appendix A and are summarized by Table 6. As detailed in Table 6, by the end of the current 
funding cycle the program completed 27 of 35 planned activities (77%). Another five activities 
(14%) are either ongoing or partially completed, two (6%) are to be carried over, and one (3%) 
has been cancelled. 

The process and outcome evaluation results for each component are presented by component in 
the next section of this report. 
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Table 6. Summary of Program Activities Status 



i Component 


Completed 


Ongoing 


Partially 

completed 


Carry 

over 


Not 

completed; 

cancelled 


Totai 1 


i Information 
ij Development and 
^ Dissemination 


9 


2 


1 


1 


0 


13 


3 Education 


8 


0 


0 


1 


0 


9 


] Alternatives 


1 


0 


0 


0 


0 


1 ; 


j Problem 
Identification and 
i (Referral) Early 
1; Intervention 


0 


1 


0 


0 


0 


1 f 


^ School/Community- 
1 Based Processes and 
i Approaches 


5 


0 


1 


0 


1 


7 1 


1 Environmental 


4 


0 


0 


0 


0 


‘ i 


I Total 


27 


3 


2 


2 


1 


35 , 
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Component I. Information Development/Dissemination 



In addition to serving as a clearinghouse and providing culturally appropriate materials, the 
program also continued its dissemination activities by participating in statewide health fairs, 
providing speaking engagements, and developing and implementing awareness campaigns such 
as Red Ribbon Week and other similar events. In addition, two on-site libraries, one at the 
campus site and one at the N^akuli site, were visited a total of 1 5 1 times. A total of 2,278 
persons were served by this component. 

Process Evaluation 

The process evaluation for this component included the following three objectives and related 
findings: 

Successfully implement at least 95% of all planned Information Development/Dissemination 
component activities. 

The milestone achievement analysis indicated that 11 of the 13 planned activities (85%) were 
completed or are ongoing, one is partially completed (8%), and one (8%) will be carried over to 
next year. 

Increase the number of materials developed by the program by 5%. 

The program has developed two entirely new products this year: the Family Laundry Kit and 
violence prevention materials for a pre-school aged E Ola Pono. Two more materials are 
nearing completion: a “Be Akamai About Violence” brochure and the “Nanakuli High Revisited” 
video. Taking into account only completed materials, this development represents no increase 
over last year’s number of two additions. 

Increase the number of materials disseminated by the program by 5% 

According to the Dissemination Form summary, over 70,200 educational materials were 
distributed during the four quarters of the grant year in comparison to 45,092 at the same time 
last year. This is an increase of 64%, or 25,108 additional materials distributed, increasing both 
the accessibility and utilization of program-developed and other materials. 

Outcome Evaluation 

To determine the effects of the dissemination of materials, PREL and KS designed a “tracer 
study” model to follow two pieces of material produced by KS. Program staff selected two 
products as representative of their work: the Lokahi Wheel and the brochure, “Re Akamai About 
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Binge Drinking." Mailing lists were compiled based on the program’s dissemination records. A 
total of 1 13 surveys were mailed out to people who had received either or both materials. As of 
early December 1999, PREL received 42 completed surveys, representing a response rate of 



37%. 



On the survey, respondents were asked to mark their choice of “strongly agree,” “agree,” 
“neutral,” “disagree,” or “strongly disagree” in relation to the following statements: 

• The materials have positively impacted student behavior. 

• The materials have contributed to alcohol, tobacco, and other drug-related services. 

• The materials have contributed to my knowledge of the harmful effects of alcohol, tobacco 
and other drugs. 

• The materials have contributed to my knowledge of Hawaiian culture. 

• The quality of materials is high. 

Survey responses are shown in Table 7 below: 



Table 1. Responses to Tracer Study of Materials 



Statement 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Don’t 

Know 


The materials have positively impacted 
student behavior. 


36% 


41% 


10% 


0% 


0% 


13% 


The materials have contributed to alcohol, 
tobacco, and other drug-related services. 


43% 


31% 


17% 


2% 


0% 


7% 


The materials have contributed to my 
knowledge of the harmful effects of 
alcohol, tobacco, and other drugs. 


39% 


33% 


19% 


2% 


0% 


7% 


The materials have contributed to my 
knowledge of Hawaiian culture. 


45% 


29% 


17% 


5% 


0% 


4% 


The quality of the materials is high. 


65% 


29% 


2% 


0% 


0% 


4% 



More than three-fourths of the respondents (77%) felt that the materials had positively impacted 
student behavior. In addition, more than 70% either agreed or strongly agreed that the materials 
contributed to their knowledge of the Hawaiian culture and the harmful effects of alcohol, 
tobacco, and other drugs. An overwhelming majority (94%) agreed or strongly agreed that the 
quality of these two materials is high. These are all good indications that the materials are 
having the effects that they were designed to elicit. 

Conclusions: Component I 

NHSDFSCP met its desired outcomes for this component. This is one of the three components 
receiving the most program emphasis. Through this component, the program is responsive to a 
frequently expressed need in the schools and communities served for prevention materials that 
are culturally appropriate for the youth of Hawai'i. The NHSDFSCP researches, produces, and 
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makes widely available materials that are particularly appealing to youth of Native Hawaiian 
ancestry. Other materials are also made available. The increasing numbers of such materials in 
circulation and the increasing use of the resource libraries are a testament to the effectiveness of 
the program in the area of information development and dissemination. 
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Component II. Education 



The educational component includes facilitating and providing technical assistance sessions, 
prevention educational presentations, and workshops. A total of 6,369 individuals were served 
by this component. 



Process Evaluation 

The process evaluation for the Education component investigates the following objective and 
related finding: 

Successfully implement at least 95% of all planned Education component program 
educational activities. 

Milestone achievement analysis showed that eight of the nine (89%) planned activities are either 
completed or ongoing, and one (11%) is to be carried over to next year. 

Outcome Evaluation 

The outcome evaluation for the Education component included three research projects: workshop 
evaluations, E Ola Pono Pre-Post-Post test evaluation, and E Ola Pono Summer School Pre-Post 
test evaluation. Results from the analysis of each of these projects are described below. 

Workshop Evaluation 

Each program’s desired outcomes, as well as several other outcomes, were measured by 
responses to workshop evaluation forms. A revised workshop evaluation measure was used this 
year to allow the collection of data relevant to stated outcomes and specific workshop topics. 
Each outcome was put in the form of a statement e.g., “Because of this workshop, I learned more 
about...”). Participants answered using a 5-point response format, ranging in value from 
“strongly agree” (5) to “strongly disagree” (1). The five statements measuring outcomes relevant 
to the Education component are presented below. Average responses to each statement, 
according to question number (i.e., 1-5) are provided by Figure 5. 

1. Increased knowledge of dangers of alcohol, tobacco, and other drug effects. 

2. Increased knowledge of dangers of violence. 

3. Less likely to use alcohol, tobacco, and other drugs. 

4. Less likely to engage in violence. 

5. Increased cultural awareness. 
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1 .uu^ 


increased knowledge 
ATOD (n=862) , 


Increased knowledge 
violence (n=390) 


Less likely ATOD use 
(n=762) 


Less likely violence 
(n=202) 


Increased cultural 
awareness (n=786) 


Series 1 


4.09 


4.10 


4.09 


4.17 


3.99 



Figure 5. Average Workshop Ratings by Education Component Statement 



1er|c 



The results show very positive overall outcomes resulting from the educational workshops. All 
of the ratings are well above the midpoint of the five-point scale. It is particularly encouraging 
that the items related to behavioral change scored high. 

Evaluation of the E Ola Pono Curriculum: Pre-Post-Post Test Analysis 

E Ola Pono (Live the Proper Way) is a substance abuse/violence prevention curriculum 
composed of thirteen lesson plans developed by the NHSDFSCP. Each lesson plan features a 
poster and related activities that teach a Hawaiian value or concept considered relevant by 
program personnel to substance abuse and/or violence prevention. The core premise of the 
curriculum is that the instruction and assimilation of these values will increase student resiliency, 
resulting in reduced drug and violence-related behaviors. 

To assess the causal influence of the curriculum on relevant student behavior, attitudes, and 
knowledge, a follow-up evaluation of the E Ola Pono (EOP) curriculum was conducted. 
Planning for the evaluation was initiated in the summer of 1997 during the curriculum planning 
stages. The evaluation design was created in collaboration with staff at NHSDFSCP in order to 
meet the directives of the Safe and Drug-Free Schools and Communities Program, which 
requires schools to “design and implement their activities based on research or evaluation that 
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provides evidence that the strategies used prevent or reduce drug use, violence, or disruptive 
behavior among youth.” 

An experimental design was selected for the evaluation of the curriculum. Using this approach, 
classes drawn from geographically varied schools in the State of Hawai‘i were randomly 
assigned within schools to function as either a control or experimental condition. 

The evaluation of the curriculum is meant to provide information corresponding to two related 
interests. First, the evaluation is intended to furnish information useful for program monitoring 
and improvement. This evaluation type is process-based in nature. This implementation 
evaluation occurred in the fall of 1998 and is described by an earlier evaluation report (PREL, 
1999). 

The second purpose of the evaluation is to provide information related to the intended outcomes 
of the curriculum, so that the degree to which these goals have been achieved can be observed. 
This evaluation is outcome-based in nature. A previous evaluation report describes the 
preliminary outcomes associated with the curriculum for both fourth and eighth grade cohorts. 
This evaluation assesses the long-term outcome of the curriculum on the fourth grade cohort 
only, by the using a delayed post-test design. Pre-, post- and post-post-test comparisons are 
made of behavioral (e.g., use of violence, use of gateway drugs), cognitive (e.g., curriculum 
knowledge), and normative (e.g., perceived norms for violence and drug-related behaviors) 
responses to a self-report instrument. The long-range impact of the EOP program is assessed by 
administering the post-post-test instrument approximately six months after the curriculum is 
provided to students. The specific methods by which the outcome evaluation was conducted are 
described below. 

Method 

The curriculum was provided to students in an experiment comparing the pre- and post-test 
scores of students quasi-randomly assigned by class to either a control or experimental condition. 
Though intended to be a true random assignment to condition, class assignment was non-random 
for at least one school, due to a strong teacher preference for assignment to the experimental 
condition. The specific nature of this experiment is described below. 

Subjects. Teachers providing instruction in the subject areas of social studies and health in the 
fourth and eighth grades of five schools were selected for participation in this study. School 
selection was based upon the following criteria: school enrollment of 50 percent or more Native 
Hawaiian students, or a significantly high number of Native Hawaiian students when compared 
against other Hawai'i public schools, and permission to conduct the study from the principal and 
teachers in each school. 

In accordance with human subject mandates, parental permission forms were distributed by 
teachers to each child present in all experimental classes. Children who returned forms 
indicating approval were selected to be in the study and were provided with pre-test (September 
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1998), and post-test (November 1998) administrations. In addition, elementary students were 
provided with an additional post-post-test assessment in May 1999. 

Students attending classes in the subject areas of social studies or health in the fourth grade of 
four elementary schools were selected for participation in this study. Two classes from each 
school were selected to participate in the study-one experimental and one control— for a total of 
four experimental and control classes. A total of 133 elementary students participated in the pilot 
test of the curricula — 68 in the experimental condition and 65 in the control condition (see Table 
8). Of this number, 68 were female (51%) and 65 were male (49%). The ethnic breakdown 
included 111 Hawaiian/part-Hawaiian (84.5%) and 22 non-Hawaiian/part-Hawaiian (16.5%). 



Table 8. Number of Subjects by School and Condition 





; - .V .y- / 'y; 


■ ' ■ ^ GoiidifiOn 


School Level 


School 


Experimental^ 


Control 


Elementary 


Hana High/ 
Elementary 


15 


17 




Waimanalo 

Elementary 

Intermediate 


14 


17 




Keaukaha Elementary 


22 


21 




Kaunakakai 

Elementary 


17 


10 




Total Sample 


68 
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E Ola Pono Process Evaluation 



The process evaluation of the E Ola Pono curriculum is of two types: the first appraises the 
degree to which the training of teachers in the curriculum is sufficient in quality and length. The 
second assesses whether an individual lesson provided to students furnishes them with the 
instruction intended. Each of these process evaluation types is described below. 

Teacher training. Teachers in the experimental condition participated in a two-day training 
session during the spring 1998 semester and a one-day refresher session in early Fall 1998. The 
training provided an overview of each lesson plan, a demonstration of each activity associated 
with the lessons, and an overview of the evaluation plan. A team of program staff members 
provided each session. The evaluations of these trainings revealed high levels of training 
satisfaction and perceived teacher preparedness. 

Teacher Questionnaire. To assess the implementation of the EOF curriculum by instructors, a 
questionnaire was provided to experimental teachers for completion after each lesson. Teacher 
responses evidenced high degrees of teacher comfort in providing the lesson, judgments of 
curriculum appropriateness, and teacher preparation in providing the lesson. Though a high 
proportion of teachers reported providing the lesson as planned and with all the materials needed, 
a substantial percentage reported having made changes to the lesson. An inspection of the 
responses to the open-ended question asking what changes were made revealed that most 
changes involved time constraints requiring a modification or deletion of a lesson component. 

Student Questionnaire. Once teachers were provided with adequate training, they introduced 
students to the E Ola Pono curriculum through individual lessons provided once or twice a week 
for 6-13 weeks. To assess student attitudes and knowledge about the curriculum, two surveys of 
students were made: the first to gain information related to students’ affective response to the 
curriculum and the second to judge their memory of its content. These measures are made to 
ensure that the curriculum is both enjoyable to students and memorable — factors presumed to be 
necessary for the curriculum to impact student behavior. 

Student responses to questions measuring lesson knowledge, application, and sharing were 
assessed for accuracy and type. Overall, student knowledge of the lessons approximately one 
week after their presentation was high. Students reported having shared the lessons with their 
fnends and family. 

E Ola Pono Outcome Evaluation 

Method 

Using the experimental design described, the outcome evaluation assesses the degree to which 
the curriculum had the intended effect on students in behavioral, cognitive, and attitudinal 
domains and is directed at assessing both the outcome and impact of the curriculum, as described 
in the following section. 
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Instrument. The pre-, post-, and post-post-test is comprised of 29 closed-ended questions. These 
questions measure the occurrence and frequency of drug- and violence-related behaviors, 
normative views regarding Native Hawaiian substance- and violence-related behaviors, and 
knowledge about Hawaiian words and concepts. The amounts of participation in, observation of, 
and fear of violence are measured by a series of questions that ask students to report the 
frequency (during the course of the last week, month, year, and lifetime) of violence-related 
experiences overall and while in school. These questions ask how many times, if any, students 
have “been in a fight where there was pushing, slapping, etc.,” “witnessed a fight,” or “feared 
being injured by others.” Seven categories for reporting frequencies of these experiences are 
provided: 0, 1, 2, 3-5, 6-9, 10-19, and 20+. 

Use of gateway drugs was measured by questions that ask students to respond to the number of 
times in their lifetime they “had alcohol to drink,” “smoked cigarettes,” and “tried marijuana.” A 
measure of students’ normative views of Native Hawaiian substance- and violence-related 
behaviors is provided by a four-part scale that queries students’ opinions regarding how many 
Native Hawaiian adults “drink alcohol,” “smoke cigarettes,” “use marijuana,” and “fight.” Five 
response categories are associated with these questions, including “almost none,” “about 25%,” 
“about half,” “about 75%,” and “almost all.” 

Students’ knowledge of Hawaiian words and phrases provided by the curriculum is also 
measured by asking students to draw a line from the 1 1 Hawaiian words and concepts taught by 
the curriculum to their English version. 

The post-post-test is a shorter form of the original pre- and post-test, with the first questions 
equivalent, and the latter ethnic identity and resiliency scales removed. The decision to simplify 
the form for the post-post-test was based on a desire to focus primarily on behavioral and 
knowledge outcomes, to exclude scales found to have ambiguous questions, and to adjust for the 
lack of significant findings related to the excluded scales in the initial post-test analyses. 

Procedure. The curriculum was provided to students, approximately two lessons a week, over 
the course of seven weeks during fall 1998. Before the first lesson, teachers distributed the pre- 
test to students. Attached to each survey was a Post-it Note® with each student’s name on it. 
Names correspond to code numbers written on the top-right comer of the survey, so that student 
identities were available to the researchers but not the teachers. Teachers were instructed to pass 
out the surveys to students by name and not to give a survey to anyone other than the person 
whose name was attached to it. If students were in the class but did not have a survey assigned, 
instructors were asked to give them a blank one, assign a three-letter string of letters to it, and 
ensure that the name corresponding to the three-letter string of letters was attached with the 
survey bundle. 

The completion of the survey took approximately 10-20 minutes. Post-test questionnaires were 
collected at the end of Lesson 13, in the same manner described above for the pre-test. Post- 
post-test questionnaires were also provided to students in the same manner four to five months 
after the completion of the curriculum. 
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Results 



Preliminary results of the curriculum evaluation are presented below. These results should be 
interpreted with caution, as the curriculum, its implementation, and the instrument measuring its 
outcome are undergoing revision as a result of this piloting phase. 

Inferential Statistics: Analysis of Variance 

Because a central interest of this study is to ascertain whether differences in levels of behaviors 
and attitudes occur for children in the differing experimental conditions, a series of repeated 
measures multivariate analysis of variance (MANOVA) were conducted to reveal whether each 
of the behavioral, attitudinal, and knowledge variables differ according to experimental condition 
over time. MANOVA is a statistical technique that compares the actual differences between the 
means of two or more groups with the differences among groups expected by chance. The 
hypothesis that the curriculum would result in an interaction effect was tested. Interactions occur 
when the effect on one factor (e, g., pre- to post-test substance abuse levels) depend upon a 
second factor (experimental condition). In this case, the interaction tested was whether 
experimental group outcomes were different than the control group from pre- to post-test. Main 
effects of period of testing (i.e., pre-, post-, and post-post) and treatment condition were also 
tested to uncover differences among the treatment groups and period of testing independent of 
curriculum effects. 

MANOVA: Violence-related behavior. An analysis of the measures of violence-related behaviors 
resulted in six statistically significant findings, including three main effects and three significant 
interactions of time and condition, as presented by Figures 6-11. 

A near-significant interaction of period of testing and treatment condition was found for the 
question ‘‘‘‘How many times, if any, have you been in a fight where there was pushing, slapping, 
etc. during the last week while in schoolT’ (F = 2.71 ; df = 2; 240, p =.068). As presented by 
Figure 6, the curriculum appeared to have an immediate effect for the experimental group, which 
was not maintained at the second post-testing. 
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Figure 6. How many times, if any, have you been in a fight where there was pushing, 
slapping, etc. during the last week while in school? 



Significant main effects of treatment condition (F = 4.86; df = 1, 125; p = .029) and period of 
testing (F = 4.24; df = 2, 250; p =.015) were found in response to the question ''How many times, 
if any, have you been in a fight where there was pushing, slapping, etc. during the last month?” 
(see Figure 7). This result indicates that the level of related fighting among the experimental and 
control groups increased equally over time, with the experimental group having higher overall 
levels. 




Experimental 

Control 



Figure 7. How many times, if any, have you been in a fight where there was pushing slapping, 
etc. during the last month ? 
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Figure 8 illustrates the statistically significant interaction of a third measure of violence: “//ow 
many times, if any, have you witnessed a fight during the last week? ” (F = 5.58; df = 2; 258, 
p =.004). While the experimental group’s related experiences were higher than those of the 
control group at pre-test, their levels reduced to converge with the control group at both post- 
testings. This result suggests an effect of the program that is immediate and maintained through 
the subsequent semester. 




As illustrated by Figure 9, a near significant main effect of period of testing was found for 
responses to the question “//ow many times, if any, have you witnessed a fight during the last 
week while in schoolT’ (F = 2.38; df = 2, 234; p =.095). This result suggests an even reduction in 
these experiences for both treatments groups. 
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Pre Post Post-Post 



Figure 9. How many times, if any, have you witnessed a fight during the last week while in 
school? 



In addition, a near-significant main effect of experimental group (F = 3.44; df = 1, 126; p = .066) 
and significant main effect of period of testing (F = 7.00; df = 2, 252; p =.001) was found for the 
question “//ow many times, if any, have you witnessed a fight during the last month?" (see Figure 
10). This finding indicates that the trends in witnessing a fight is the same for both the 
experimental and control groups — with an increase in experiences followed by a decrease — and 
that the experimental group had higher overall levels of this experience at each testing period. 




Pre Post Post-Post 



Figure 10. How many times, if any, have you witnessed a fight during the last month? 
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As illustrated by Figure 1 1, a statistically near-significant interaction was discovered for the final 
measure of violence “//ow many times, if any, have you feared being injured by others during the 
last week? ” (F = 2.49; df = 2, 260; p =.085). While the experimental group had initially higher 
levels of this variable than those in the control condition, their levels dropped at subsequent 
testings to become nearly equivalent to those in the control group. 




ANOVA: Drug-related behavior. As illustrated by Figure 12, analyses investigating the 
potentially significant infiuence of the curriculum on drug-related behaviors resulted in a 
significant interaction for the question “//ow many times, if any, have you had alcohol to drink in 
your lifetime? ” (F = 8.31; df = 2, 256; p < .000). Because a decrease in the reported incidence of 
drinking alcohol is cumulative and cannot decrease over time, however, these results suggest 
some degree of response bias among the experimental sample. The control group responses 
revealed a trend towards steady increase over time. 
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Responses to a second gateway drug question revealed a near-significant main effect of period of 
testing on the question ’‘‘‘How many times, if any, have you smoked cigarettes in your lifetime?” 
(F = 2.87; df = 2, 262; p =.059), indicating a trend over time for increasing use, across groups 
tested (see Figure 13). 




Figure 13. How many times, if any, have you smoked cigarettes in your lifetime? 



No significant main effects or interactions were found for the question asking about lifetime use 
of marijuana, indicating that decisions to engage in this behavior were unaffected by either the 
time course of the testing period or the presence of the curriculum. Because the range in the 
incidence of this behavior is slight for this age group, it is likely that the variance was 
insufficient to find significant effects. 




34 



PACIFIC RESOURCES FOR EDUCATION AND LEARNING 2000 



Page 30 



nf M^f ’ Perceptions. Responses measuring students’ perception of the proportion 

of Native Hawaiians who engage in various behaviors were also analyzed. A significant main 
effect of period of testing as discovered for the question “/« your opinion, how many Native 
Hawaiian adults drink alcoholT (F = 5.87; df = 2, 260; p =.003) (see Figure 14) ^ 
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Finally, a significant main effect of period of testing was also revealed for the question “/« your 
opinion, how many Native Hawaiian adults use marijuanaT (F = 4.82; df = 2, 256; p =.009) (see 
Figure 16). 




Figure 16. In your opinion, how many Native Hawaiian adults use marijuana? 



The nearly equivalent results of each of these normative perception measures indicates a steady 
trend over time for increased perceptions of use of cigarettes, alcohol, and marijuana among 
children in reference to Native Hawaiian adults. The trend for the final question measuring 
perceived norms for fighting did not vary significantly across treatment group or time period. 

ANOVA; Knowledge. Significant interactions between experimental condition and time period 
were found, as depicted by Figure 17 (F = 84.05; df = 2, 262; p < .000). These results indicate a 
very robust effect of the curriculum on student memory for the Hawaiian words and phrases upon 
which the curriculum is based. 
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Experimental 

■— □ ■—Control 



Figure 1 7. Match the following Hawaiian words and phrases to their English version. 




Overall findings of E Ola Pono Evaluation 

There appears to be an immediate effect of the curriculum on each of the major violence-related 
measures, including participation in, observation of, and fear of violence. This effect appears, 
however, to be modest and inconsistent. For example, participation in monthly fights showed a 
trend toward increase, regardless of condition, while weekly fights in school were found to 
decrease for those in the experimental condition. These effects, though statistically significant or 
approaching significance, are relatively low in intensity. The observation of fights over the 
course of the week preceding testing showed a consistent trend towards decrease for the 
experimental group, while the pattern of the control group did not. Monthly measures of fight 
observations did not repeat this pattern, however. These results again suggest the potentially 
favorable effect of the curriculum on violence-related experiences, though one that is modest and 
not consistently observed. Finally, the experimental group showed lower levels of fear of 
violence subsequent to the curriculum intervention, relative to pre-test levels, even though they 
had higher initial baseline levels. The consistent finding overall was that the experimental group 
had higher baseline levels of violent experiences than the control condition, and that these 
differences vanished after the initiation of the curriculum. 

Results of the gateway drug use responses reveal a predominant trend toward increase over time 
in alcohol and cigarette use. While these effects were small in degree, they speak to the need for 
interventions beginning as early as the fourth grade. No significant main effects or interactions 
were found for the question asking about lifetime use of marijuana, indicating that decisions to 
engage in this behavior are unaffected by either the period of testing or the presence of the 
prevention curriculum. Because the range in the incidence of this behavior is slight for this age 
group, it is likely the variance was insufficient to find significant effects, regardless of program 
effectiveness. 
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There appears to be no effect of the curriculum on normative perceptions of Native Hawaiian use 
of cigarettes, alcohol, and marijuana, and engagement in fighting. However, trends in the data 
from pre- to post-post testing evidence a consistent increase in these perceptions over time, 
indicating a maturation process by which children are becoming more aware of these behaviors 
and their prevalence. 

Finally, the program was found to have a robust effect on student learning and retention of 
Hawaiian values. 

Implications 

The results of the evaluation of the E Ola Pono curriculum are suggestive of an effect of the 
program on violence-related experiences that is immediate and sustained over time. However, 
statistical significance speaks only to the non-randomness of the effect and does not imply its 
consequence. That is, statistical significance is not equivalent to practical significance; the effect 
may be decipherable but unimportant. At this stage in the development and implementation of 
the curriculum, the effects on student behavior appear to be existent, but small. 

The finding of a statistically significant effect on behavior is notable nonetheless, and provides 
information useful toward the development of recommendations for future practice. For 
example, the effect of the curriculum could be bolstered potentially by a greater repetition of its 
component lessons. This repetition might take the form of a longer intervention, or booster 
periods following the initial curriculum provision. 

A need for instrumentation revision was evident after the pre-test data collection period. Teacher 
interviews revealed unanticipated problems with the instrument including its length, associated 
student fatigue and apathy, and question ambiguity. A revised instrument was pilot-tested in the 
summer during a summer school application of the E Ola Pono curriculum. This instrument, 
complete with new and revised scales, is an improvement over the previous one, but is not yet 
finalized. 

A further implication of the E Ola Pono evaluation is the call for continued research of the 
curriculum, including a better instrument — one that is shorter, easier to complete, and less 
ambiguous — and the inclusion within the overall research design of exploratory and qualitative 
data, such as anecdotal reports, teacher checklists, and non-participant observations. 

Finally, the trend data provided by this study can provide hints towards the effects of maturation 
on the incidence of gateway drug experiences, and normative perceptions of gateway drug use. 
This type of information is useful for needs assessment purposes and for its application toward 
targeting intervention populations. 
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Findings from E Ola Pono Summer School Pre-Post 



The E Ola Pono summer school curriculum for both the Keiki and ‘Opio groups was evaluated 
to test it’s effects on the occurrence and frequency of drug- and violence-related behaviors, 
normative views regarding Native Hawaiian substance- and violence-related behaviors, 
knowledge about Hawaiian words and concepts, resiliency, and lessons. The evaluation design 
was non-experimental; no control group was tested. A pre- and post-test was provided to 
students undergoing the curriculum at the beginning and completion of a five-week summer- 
school course, and these measures were compared using a series of paired t-test analyses. 

The only significant effect found over the course of the one month from pre- to post-testing was 
the effect of period of testing on the “Knowledge” variable (Keiki: t = 11.15, two-tailed; df = 15; 
p < .000; ‘Opio: t = 4.66, two-tailed; df = 17; p < .000), measuring memory for the Hawaiian 
words and concepts around which each lesson is based. Knowledge of the material presented by 
the course was well retained by both Keiki and ‘Opio groups (see Figures 18 and 19, 
respectively). 




Figure 18. Match the following Hawaiian words and phrases to their English version: Keiki 
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The lack of a significant effect of the curriculum on the other variables of interest may be due to 
instrumentation problems. For example, because the instrument allows only a “Yes” and “No” 
format for responding, very little variance due to a restricted range of responses resulted. In 
addition, the short interval of time available to measure differences in attitudes and behavior 
added to the low variance. This negatively effects the power of statistical tests to discern 
differences. This complication of the design, coupled with a low sample size and high 
homogeneity of population, reduced the probability of significant findings. 

Conclusions: Component II 

Education remains the cornerstone of the NHSDFSCP. The workshop evaluation forms provide 
evidence that the participants feel that the experience had an impact on them, on their knowledge 
and beliefs, and on the decreased likelihood of their being involved in both alcohol and other 
drugs and in violent behaviors. The E Ola Pono evaluation shows preliminary evidence of 
impact on students’ knowledge of Hawaiian concepts and values as well as reduced involvement 
in violent behavior for elementary school students. 
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Component III. School/Community-Based Processes and Approaches 



Under this component the program works with community groups and agencies, provides 
capacity building training in prevention skills, and assists with educational components of after- 
school activities for students. Through this component the program served 4,703 individuals 
including students, faculty/staff, and adults. 



Process Evaluation 

The process evaluation objective for this component is: 

Successfully implement at least 95% of all planned School and Community-Based component 
program educational activities. 

A total of five of the seven (71%) activities intended for this component were completed, one 
(14%) was partially completed, and one (14%) has been cancelled. 



Outcome Evaluation 

As with Education component activities, a workshop evaluation form was used to assess the 
outcome of the training provided in this component. The five statements measuring outcomes 
relevant to the School/Community-Based component are presented below. Average responses to 
each statement, according to question number (i.e., 1-5) are provided in Figure 20. 

1. Increased knowledge of dangers of alcohol, tobacco, and other drug effects. 

2. Increased knowledge of dangers of violence. 

3. Less likely to use alcohol, tobacco, and other drugs. 

4. Less likely to engage in violence. 

5. Increased cultural awareness. 

These results show very positive overall outcomes resulting from School/Community-Based 
component workshops. All of the ratings are well above the midpoint of the five-point scale. 
The highest rating was 4.30 in response to the statement, “This workshop increased my cultural 
awareness.” All of the items on alcohol, tobacco, and other drugs, and violence scored at about 
the same level— 4 on a 5-point scale. 
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Conclusions: Component V 



In summary, this component represents a major strategic thrust of the NHSDFSCP. The program 
is very effective at collaboration with similar agencies and within specific communities. This 
affiliative approach is both cost efficient and highly valued within the Hawaiian social system. 
This component has also stimulated the funding of research upon which to rationally develop and 
apply program plans. Finally, in terms of increasing the knowledge and skills of teachers, those 
who have been the direct recipients of curriculum training are overwhelmingly favorable about 
the training and its potential for having a positive impact on students. 




Figure 20. Average Workshop Ratings by School/Community-Based Component Desired 
Outcome 



Other Components 



The other components are: 



• Alternatives 

• Problem Identification and (Referral) Early Intervention 

• Environmental 
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Although not the program’s main emphasis, a total of 89 persons were served under these 
components. A total of six of the six (100%) activities planned for these components were 
completed or are ongoing. 
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In the logic model used by the NHSDFSCP, “impact” means long-term outcomes, as opposed to 
the short-term, immediate outcomes reported as program objectives. Impact refers back to the 
program’s mission, to “reduce alcohol, tobacco and other drug (ATOD) use and involvement in 
violence by Native Hawaiian youth.” Each year, PREL monitors statewide data on Native 
Hawaiian student substance use and violence-related behavior. 

These large-scale data sets reveal trends that are broad and slow to change. Trends at this level 
are affected by many factors. Clearly one program cannot claim to be responsible for them. 
Nonetheless, as the NHSDFSCP achieves its immediate objectives, it is contributing to success 
at the statewide level. Therefore it is critical to monitor and report on impact at this level. 

Several new surveys and studies have been completed and provide data for impact analysis. 

The Hawai‘i Poll of Teen Attitudes 



This study was very recently completed by Ward and Associates and was reported in the 
Honolulu Advertiser (December 12 - 15, 1999). Overall, the study found teens to be happier, 
more optimistic, and more well off than most adults had thought. 

Eighty-four percent of Hawaii’s teenagers feel safe in school most of the time.. .Seventy-one 
percent said they feel “cared about.” Sixty-seven percent said they were happy. Fifty-six percent 
said they were “fortunate” (Honolulu Advertiser, December 12, p.l). 

Thirty-eight percent of adults said violence in schools was a very serious problem, compared to 
only 13 percent of teenagers, and about 67 percent of adults believed drugs and alcohol were “a 
very serious problem” for teens, compared to only 29 percent of the teenagers. The 
disconnection might occur because parents don’t realize that their children already have faced 
decisions over drug or alcohol use, some as early as junior high school, as some teenagers said. 
(12/12/99, p. A2). 

There were very some positive signs among Hawaiian students when ethnic differences were 
considered: 

• The Hawaiian teenagers were most likely to say they were cared about (75%), happy (74%), 
and fortunate (64%) most of the time (12/14/99, p.A2). 

• They are looking ahead. Some 84% of Hawaiian students, second only to Caucasian students 
at 86%, say they spend a great deal or a fair amount of time thinking about life after high 
school. 

» Hawai‘i is important to them. Sixty-six percent of Hawaiian students believe they will live in 
Hawai‘i after school, while only 27 percent of Caucasian students believe they will. 
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o They value education. Hawaiian teens reported that they feel pressure to get good grades 
(74%, second only to Japanese students at 79%). 

The results support the directions chosen by the NHSDASCP: to emphasize Hawaiian culture in 
the educational setting and to provide protective factors early, before confrontations with ATOD 
and violence, which many teens feel they have already gone through. It is encouraging indeed 
that the profiles of Hawaiian teens are as positive as those of other ethnic groups. 

1998 Hawai‘i Student Alcohol and Drug Use Study 

In the spring of 1998, the Hawai'i Student Alcohol and Drug Use Study was conducted by the 
Alcohol and Drug Abuse Division of the Department of Health in Hawai'i (Klingle & Miller, 
1999). It follows previous similar surveys conducted in 1987, 1989, 1991, 1993, and 1996. The 
survey was administered to over 25,000 students in the sixth, eighth, tenth, and twelfth grade 
levels enrolled in 204 public schools and 44 private schools across the state. All students who 
completed a questionnaire received consent to do so from their parents. Because the Native 
Hawaiian Safe and Drug-Free program focuses its services on Hawaiian students, survey results 
for Hawaiian students are given here. 

Native Hawaiians reported higher use of alcohol, cigarettes, and other drugs than did other ethnic 
groups. The survey found that the level of substance use decreased for all ethnic groups when 
compared to levels reported by students two years ago. Nevertheless, the level of drug use 
reported on the 1998 survey was higher than that reported on the 1993 survey. These findings 
are consistent with the conclusion that, while the extent of drug use by Hawaii’s students has 
lessened somewhat from prior levels, more remains to be done for all students. 

The study identified 14 factors that heightened students’ risk of drug abuse and 6 factors that 
inhibited drug use. (Examples of risk factors identified in this study are availability of 
substances and exposure to friends ' substance use. Examples of protective factors identified in 
this study are disapproval of substance use by peers and parental sanctions on substance use.) 
When the frequency of these risk and protective factors was tabulated for each student, it was 
found that Native Hawaiians, as a group, reported the presence of more risk factors and fewer 
protective factors than did other ethnic groups in Hawaii. 

Based on the results of the survey, the Alcohol and Drug Abuse Division noted that most 
students did not think the substance education programs at their schools were good. Students 
gave similarly low ratings to the capacities of their families to impart meaningful substance 
abuse education. In light of the protection against drug use invitations conferred by family 
substance education, the division claimed “improvements in alcohol and drug education must 
occur at both the school and family level” (p. 18). Consistent with this recommendation, the 
NHSDFSC program is currently developing a school-based version of their E Ola Pono program 
that includes parents. 
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A related recommendation was that “comprehensive school-based substance abuse prevention 
programs must begin no later than the fourth grade” (p. 27). This year, the NHSDFSC program 
is emphasizing development of its program for fourth graders and is developing “boosters” 
against substance use to occur thereafter at strategic points through middle and high school. 

Another recommendation made by the Alcohol and Drug Abuse Division is to widely distribute 
information about the availability of treatment and counseling services. Through its Problem 
Identification and Early Intervention component, the NHSDFSC program informs students about 
where and how they can seek help for substance abuse. 

Client Feedback 



Along with statewide surveys and data tracking, the program keeps a file of letters and notes 
fi'om clients, providing an unsolicited source of impact data. Appendix B contains a more 
complete record of the client and student feedback file. Here are only a few examples: 

• This program is a great opportunity to use our native Hawaiian ancestral values and 
incorporate into teaching about a safe and drug free environment. 

• We appreciate your efforts and also the materials that you provided the participants on the 
Native Hawaiian Drug Free Schools and Communities program. 

• The material you presented was very informative and helpful to the students who are in 
service and work with Native Hawaiian families and teenagers facing problems relating to 
drugs, family violence and child abuse. 

• The experience will help me to do better work and to be more effective in my community. It 
is so very valuable to learn about what is happening not only on the other islands of Hawai‘1, 
but also on the mainland and other places in the world. I came away from the conference 
energized and full of ideas for the future. 

The feedback reviewed by the PREL evaluation team was uniformly positive, with specific 
references to how the program’s services have impacted the work of teachers and the attitudes 
and behaviors of students. 
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Commemdatioms and Recommendations 



Commendations 



1. The NHSDFSCP Director is commended for her firm commitment to sound program 
planning based on outcome data and needs assessment results. She has maintained the use of 
the Logic Model approach to program and evaluation planning. Each year the program has 
been able to set clearer outcome objectives, and this year they were able to meet three out of 
eight of those for which data were available at the end of the program period. Those not met 
were within 8 percentage points of criterion. This is more impressive in that baseline rates 
for these particular outcome targets had not been previously established. 

2. The program is commended for its application of needs assessment information. From the 
beginning the NHSDFSCP has been guided by continuous needs assessment efforts. These 
have included monitoring Native Hawaiian student and adult involvement in alcohol, 
tobacco, and other drug (ATOD) use and violence as well as community-felt needs. Needs 
assessment provided the information that allowed the program to focus on those schools and 
communities with a high number and proportion of Native Hawaiians. 

3. One of the first clearly defined needs was for information on ATOD use and violence to be 
presented in forms that were culturally sensitive and locally relevant. Accordingly, the 
program spent much time in developing new materials based upon Hawaiian or local cultural 
themes. The program is commended for the cultural relevance of these materials. 

4. The Alcohol and Drug Abuse Division notes that “improvements in alcohol and drug 
education must occur at both the school and family level” (Klingle & Miller, 1999, p. 18). 
Consistent with this recommendation, the NHSDFSC program is commended for its 
development of a school-based version of their E Ola Pono program that includes parents. 

5. A related recommendation by the Alcohol and Drug Abuse Division is that “comprehensive 
school-based substance abuse prevention programs must begin no later than the fourth grade” 
(Klingle & Miller, 1999, p. 27). The NHSDFSC program is emphasizing development of its 
program for fourth graders and is developing “boosters” against substance use to occur 
thereafter at strategic points through middle and high school and is commended for this 
initiative. 



Recommendations 



1 . It is recommended that the program carry out a more intensive, observational assessment of 
the E Ola Pono curriculum. This should be done using program developers as teachers or in 
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direct support of teachers so that they have a better on-site sense of the implementation of the 
curriculum. 

2. Now that the E Ola Pono curriculum has been pilot-tested and refined, it is recommended 
that the program initiate a longitudinal study of its long-term effects on students. The 
limitations of a one-semester intervention and pre and post measurements are obvious. 
Whether the program provides long-term protective or preventive benefits can only be seen 
by following carefully constructed samples of students through their high school years - 
possibly even beyond. Long- term outcomes to be followed might include school attendance, 
grades and achievement, and participation in extra-curricular activities. 

3. The effect of the E Ola Pono curriculum should be bolstered by a greater repetition of its 
component lessons. This repetition might take the form of a longer intervention or periodic 
booster sessions to critical program features following initial exposure to the curriculum. 

4. A need for instrumentation revision was evident after the pre-test data collection period. 
Teacher interviews revealed unanticipated problems with the instrument including its length, 
associated student fatigue and apathy, and question ambiguity. A revised instrument was 
pilot-tested in the summer during a summer school application of the E Ola Pono 
curriculum. This instrument, complete with new and revised scales, is an improvement over 
the previous one, but is not yet finalized. Future instrumentation improvements should 
attempt to ensure that assessments are culturally appropriate and understandable, given 
existing literacy limitations of students. 

5. A further recommendation for the E Ola Pono evaluation is the use of qualitative research of 
the curriculum. For example, there should be the inclusion within the overall research design 
of more exploratory and qualitative data, such as anecdotal reports, teacher checklists, and 
non-participant observations. 

6. The trend data provided by this study provide information regarding the effects of maturation 
on the incidence of gateway drug experiences, and normative perceptions of gateway drug 
use. This type of information should be applied toward program needs assessment and for its 
application toward targeting intervention populations. 

7. A recommendation made by the Alcohol and Drug Abuse Division is to widely distribute 
information about the availability of treatment and counseling services. Through its 
Information Development and Dissemination component, the NHSDFSC program should 
continue to inform students about where and how they can seek help for substance abuse. 
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Appendix A: Milestone Achievement Analysis 
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Kamehameha Schools 
Safe and Drug Free Program 

Milestone Analysis of Program Goals and Objectives 
Year Four (3/1/99 - 2/29/2000) 



Mission: 

Reduce alcohol, tobacco and other drug (ATOD) use and involvement in violence 
among Native Hawaiian Youth. 



Goals: 

1 ) Increase knowledge of the dangers of ATOD use, and of violence, and 
increase cultural awareness by increasing the availability and use of 
culturally appropriate prevention materials. 

2) Reduce ATOD use and violence-related behaviors among participating 
Native Hawaiian youth by providing direct, culturally appropriate 
educational services at high need schools and communities. 

3) Build the capacity of schools and communities to provide safe and 
drug-free activities for Native Hawaiian youth and adults. 



Targeted Native Hawaiian Schools and Communities: 

Kamehameha Schools: Kapalama Campus, Maui Campus, Hilo Campus, 
Kamehameha Preschools. 

Wai‘anae Coast: Nanakuli Elementary, Nanakuli Intermediate/High, Nanaikapono 
Elementary. 

Waimanalo Coast: Blanche Pope Elementary, Waimanalo Elementary/Intermediate, 
Kailua High 

Kaua‘i: Kapa‘a Middle School, Kekaha Elementary. 

Maui: Hana High/Elementary. 

Hilo: Keaukaha Elementary, Hilo Intermediate. 

Moloka'i: Kaunakakai Elementary, Maunaloa Elementary. 

Lana‘i and other Native Hawaiian schools and communities. 
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MILESTONE ANALYSIS 
March 1, 1999-February 29, 2000 



Component I. Information Development/Dissemination 



A." Serve as a clearinghouse and maintain^itwo information/" 
prevention resource centers appropriate for Native 
Hawaiians 




1. To acauire and replace locallv produced culturallv 
appropriate educational materials on violence and 
substance abuse prevention, resources and referral. 


Completed/ 

Ongoing 


2. To acauire and replace other educational materials 
on substance abuse, violence prevention, needs 
assessment, research, resources and referral. (Note: 
focus on violence and school safety). 


Completed/ 

Ongoing 


3. To develop and maintain a web site for the program. 
B. Develop five and complete two new curriculum/research 
products 


1/2000 


1. To develop an informational brochure on violence 
prevention, “Be Akamai About Violence”. 


Phase I 
Completed/ 
Phase II Carry 
Over 


2. Maintain an updated bibliography on Native Hawaiian 
substance abuse and violence (Updated yearly): 

• Research bibliography 

♦ Web-site links 


Completed 


3. To develop the supplemental violence prevention 
lesson plans for the E Ola Pono Curriculum. 


Carry Over 


4. To develop age appropriate activities for pre-school 
children. 


Completed 

11/99 


5. To develop and Pilot E Ola Pono activities for 
adults. 


Completed 

10/99 


6. To complete the video tape Nanakuli High Revisited. 


Carryover 


7. To develop a project dissemination kit for the “Family 
Laundry” video. 

C. Disseminate educational materials and other information 


Completed 

9/99 


1. To distribute oroaram develooed educational 
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Component 1. Information Development/Dissemination 


materials. 




2. To distribute educational materials developed by 
others to schools and Community partners. 


Ongoing 


3. To distribute the videotape Nanakuli High Revisited. 


Carryover 


4. To participate in (2) information dissemination 
activities (i.e., health fairs, Lana'i Family Fair, 
information booths, etc.) 


Completed 




54 



PACIFIC RESOURCES FOR EDUCATION AND LEARNING 2000 



Page 50 



MILESTONE ANALYSIS 
March 1, 1999-February 29, 2000 



Component II. Education 






A. To facilitate and/or provide technical assistance for three 
prevention education projects. 




1 . To support after school programming at 

Kamehameha Schools and other selected schools 
and Communities (i.e., KES Kapalama, Hui Malama 
0 Ke Kai, HKO, etc.). 


Completed/ 

Ongoing 


2. To support substance abuse and violence prevention 
type programs at Kamehameha Schools and other 
selected schools and Communities (i.e., KIS Study 
Hall, Kaunakakai Peer Ed., Moloka'i Video Class, 
etc.). 


Completed/ 

Ongoing 


B. To provide prevention education presentations and 
workshops. 


Ongoing 


1 . To provide 20 prevention education 
presentations/workshops that enhance 
comprehensive program initiatives in targeted 
schools and Communities. 


Completed/ 

Ongoing 


2. To provide 10 prevention education 

presentations/workshops characterized as singular 
learning experiences. 


Completed/ 

Ongoing 


C. To provide Parenting/Family Management services. 
(Services here must be attached to other program 
initiatives) 




1 . Conduct workshops statewide for parents of students 
at Kamehameha Schools or other school/Community- 
based sites (i.e., KES: Kapalama, Maui, Hilo; KS 
Preschools, Open Minds, etc.). 


2/2000 


2. Conduct two family based retreats with at-risk 
students at Kamehameha Schools or other 
school/Community-based sites (i.e., KIS, Hui Malama 
0 Ke Kai, Moloka'i, etc.). 


1/2000 


3. Support Family Literacy programs in targeted 
school/Community based sites (e.g., Waimanalo 
Adult Literacy Program). 


Not completed/ 
Revised 
Objective 
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Is this the one that was “cancelled” or the refusal skills 
study? the report discusses “one” 




D. To implement the £ 0/a Pono curriculum. 




1 . Provide technical assistance for the implementation 
of the EOP curriculum (i.e., KES Statewide, ADAD, 
HHA, ALU LIKE, etc.). 


Completed/ 

Ongoing 


2. To implement E Ola Pono at six school/Community 
sites statewide (i.e., KES Statewide, etc.). 


Carryover 

6/2000 


3. To replicate the E Ola Pono Summer School 

Curriculum at two sites (i.e., Kapa‘a Middle School, 
Keaukaha Elem.). 


Completed 

7/99 
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MILESTONE ANALYSIS 
March 1, 1999-Febroary 29, 2000 



Component III. Alternatives 


Planned Activities (Process Ot^ectives); ^^^^^ 


Due J 


A. To support safe and alcohol, tobacco and other drug free 
events statewide. 


2/00 



Component IV. Problem Identification and (Referral) Early Intervention 


Planned Activities (Process Objectives) 


Due 


A. To provide resource and referral services (e.g., 
information, assessment, early intervention, treatment). 


Ongoing 



Component V. School/Community Based Processes and Approaches 




^l«|;:Pfannedl^tjyitjes 






A. To provide and participate in Community/school training 
services. ‘ 




1 . To conduct six E Ola Pono curriculum training, 
statewide, and mainland. 


Three Completed/ 
Three 

Carried Over 


2. To participate/fund olannino and orooram 

development in targeted Native Hawaiian schools and 
communities. 


Completed/ 

Ongoing 


3. To provide/fund activities in taroeted Native Hawaiian 
schools and communities. 


Completed/ 

Ongoing 


4. To provide/fund staff development in taroeted Native | 
Hawaiian schools and communities. 


Completed/ 

Ongoing 


B. To work in collaboration with other 
agencies/pfograms/schools/Communities that serve Native 
Hawaiians to plan, implement, or fund prevention efforts (i.e., 
HI-NET, HACDACS, Ke Ola Mamo, Ola Pono Awards, 
Ho'omau Ke Ola etc.). 


Ongoing 


C. To participate in needs assessment activities and 
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services. 

. 






1 . To conduct a study on culturally appropriate refusal 
skills. 

Or is this the one that was “cancelled”? I like revised the 
objective better than “cancelled” — what do you think? 


Not completed. 
Revised 
Objective 


2. To participate/fund needs assessment in targeted 
Native Hawaiian schools and communities. 


Completed/ 

Ongoing 
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MILESTONE ANALYSIS 
March 1, 1 999°February 29, 2000 



Component VI. Environmental 


Planned-Activities;(Process.Objectives)* 


Due. 


A. To provide technical assistance on environmental impact 
related issues to Communities/schools serving Native 
Hawaiians. 


Completed/ 

Ongoing 


B. To develop and implement awareness campaigns such as 
Red Ribbon Week and other similar events. 


2/2000 


C. To implement a statewide ATOD awareness campaign. 


Completed 

10/99 


D. To assist in the development and implementation of 
statewide prevention application systems (i.e., Hawai'i Center 
for the Application of Prevention Technology). 


Completed/ 

Ongoing 



03/20/2000 
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Appendix B: Client Feedback 
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